CTA-aware Prefetching for GPGPU

Several studies have been proposed to adopt memory prefetching schemes to reduce performance impact of long latency memory operations in GPUs. By leveraging a simple intuition that the consecutive warps are likely to have spatial locality, prior approaches prefetch two or four consecutive cache lines when there is a cache miss. Other approaches predict striding accesses by detecting base address and stride value from each warp’s load address history. Warp-based load prediction works well when a load instruction is repeatedly executed in a loop. But to exploit parallelism, GPU kernels favor creating a massive number of threads rather than sequential loop codes. In this paper, we exploit the observation that all the threads generated from a kernel execute the same code segment. When threads are grouped into cooperative tread arrays (CTAs), each thread uses thread id and CTA id to identify the data that it operates on. Thus the stride values for loads among warps within a cooperative tread array (CTA) can be easily detected and this stride value can be applied across all the CTAs in the kernel. However, the starting base address accessed by the first warp in a CTA is difficult to predict since that starting address depends on how the CTAs are created by the application programmer. Hence, we propose to first compute the base address of a load in each CTA by using a leading warp. The leading warp of each CTA is executed early by pairing it with warps from currently executing CTA. Thus our proposed CTA-aware prefetch predicts all the trailing warps’ load addresses by first computing the base address early through a leading warp and then using the stride value. Through simple enhancements to the existing two-level scheduler, prefetches can be issued sufficiently ahead of time before the demand requests. CTA-aware prefetch predicts addresses with over 94% accuracy and is able to improve performance by 5.4%.